Concerning the problem that in complex and non-cooperative situations the number of matching feature pairs and the accuracy of feature matching results in video stitching can not meet the requirements of subsequent image stabilization and stitching at the same time, a method of constructing matching model to match features accurately after feature points being scored by grayscale tower was proposed. Firstly, the phenomenon that the similiar grayscales would merged together after grayscale compression was used to establish a grayscale tower to realize the scoring of feature points. Then, the feature points with high score were selected to establish the matching model based on position information. Finally, according to the positioning of the matching model, regional block matching was performed to avoid the influence of global feature point interference and large error noise matching, and the feature matching pair with the smallest error was selected as the final result of matching pair. In addition, in a motion video stream, regional feature extraction could be performed by using the information of previous and next frames to establish a mask, and the matching model could be selectively passed on to the next frame to save the computation time. The simulation results show that after using this matching model based on grayscale tower score, the feature matching accuracy is about 95% and the number of matching feature pairs of the same frame is nearly 10 times higher than that of the traditional method. The proposed method has good robustness to environment and illumination while guaranteeing the matching number and the matching accuracy without large error matching result.